Background-foreground module for video analysis system

ABSTRACT

Embodiments of the present invention provide a method and a module for identifying a background of a scene depicted in an acquired stream of video frames that may be used by a video-analysis system. For each pixel or block of pixels in an acquired video frame a comparison measure is determined. The comparison measure depends on difference of color values exhibited in the acquired video frame and in a background image respectively by the pixel or block of pixels and a corresponding pixel and block of pixels in the background image. To determine the comparison measure, the resulting difference is considered in relation to a range of possible color values. If the comparison measure is above a dynamically adjusted threshold, the pixel or the block of pixels is classified as a part of the background of the scene.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims benefit of U.S. Provisional Patent ApplicationSer. No. 60/975,565 filed Sep. 27, 2007. This application relates tocommonly assigned, co-pending U.S. patent application Ser. No.12/028,484 filed Feb. 8, 2008, entitled “Behavioral Recognition System”(Atty. Docket No. BRS/0002), which are both herein incorporated byreference in their entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

Embodiments of the present invention generally relate to analyzingrecorded video, and more particularly to analyzing a stream of videoframes to generate a background/foreground image of a scene depicted inthe video frames.

2. Description of the Related Art

Some currently available video analysis systems include video contentanalysis capabilities. Such systems may be configured to analyze streamsof video frames, whether recorded or in real-time, to detect abnormalbehaviors or suspicious activities. However, many such video analysissystems lack efficiency in processing video content. For example, whilethe importance of isolating background and foreground images has beenacknowledged, many of the video analysis systems fail in implementation.Furthermore, currently available video analysis systems are ofteninadequate in maintaining the background/foreground classification asobjects move about a scene over time. Moreover, maintaining accuratebackground/foreground classifications when environmental conditions ofthe scene change has also been a challenge. For example, lightingchanges over time, clouds casting shadows, or a car headlight in a darkparking garage can all disrupt the background/foreground classificationprocess.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide a method and a system foridentifying a background/foreground of a scene. One embodiment of theinvention includes a method for identifying a background of a scenedepicted in a sequence of frames. The method generally includesreceiving a current video frame of the sequence of frames. The methodfurther includes determining a comparison measure for a block of pixelsof the current video frame. The comparison measure is determined using arange of possible values of color-characteristics of the block ofpixels, color-characteristic values of the block of pixels in thecurrent video frame, and color-characteristic values of a correspondingblock of pixels of a background image. The block of pixels may includeone or more pixels. The method further includes classifying the block ofpixels as the background of the scene when the comparison measure isabove a pre-defined comparison measure threshold and classifying theblock of pixels as a foreground of the scene otherwise.

Another embodiment of the invention includes a method for compensatingfor lighting darkness in a scene depicted by a sequence of video framesreceived from a video input source. This method may include buffering aplurality of video frames of the sequence of video frames and receivinga raw video frame of the sequence of video frames. Upon determining thata lighting level of at least one region of the raw frame is below aspecified threshold, the method may also include generating a currentvideo frame by assigning a compensated color-characteristic value toeach pixel included in the at least one region. The compensatedcolor-characteristic value for each respective pixel is calculated byaveraging color-characteristic values of a corresponding pixel sampledfrom each of the buffered plurality of video frames. Once generated, thecurrent video frame may be provided to a background-foreground moduleconfigured to identify which pixels in the current frame depict aportion of a background of the scene.

Still another embodiment includes a method for identifying stalebackground pixels in a background model of scene depicted in a sequenceof video frames. This method generally includes receiving a raw videoframe of the sequence of video frames, classifying a first set of one ormore pixels of the raw video frame as depicting a portion of abackground of the scene, and classifying a second set of one or morepixels of the raw video frame as depicting a portion of a foreground ofthe scene. For at least a first pixel classified as depicting a portionof the foreground of the scene, this method also includes (i) selectinga background pixel in the background image corresponding to the firstpixel and a plurality of pixels in the background model neighboring thebackground pixel, (ii) determining a plurality of changes betweencolor-characteristic values of the plurality of pixels in the backgroundimage and respective color-characteristic values of a correspondingplurality of pixels in a video frame previously used to update thecolor-characteristic values of the background pixel, and (iii) upondetermining a difference between the change and the plurality of changesis within a specified threshold, classifying the background pixel in thebackground image as a stale background pixel. The method may furtherinclude updating the color-characteristic values of the background pixelin the background image using the color-characteristic values of thefirst pixel.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features, advantages, andobjects of the present invention are attained and can be understood indetail, a more particular description of the invention, brieflysummarized above, may be had by reference to the embodiments illustratedin the appended drawings.

It is to be noted, however, that the appended drawings illustrate onlytypical embodiments of this invention and are therefore not to beconsidered limiting of its scope, for the invention may admit to otherequally effective embodiments.

FIG. 1 illustrates components of a behavior-recognition system,according to one embodiment of the present invention.

FIG. 2 illustrates components a background-foreground module, accordingto one embodiment of the present invention.

FIGS. 3A-3B are flow diagrams illustrating a method 300 for identifyinga background of a scene depicted in a stream of video frames, accordingto one embodiment of the invention.

FIG. 4 is a flowchart illustrating a method for identifying a backgroundof a scene using a comparison measure, according to one embodiment ofthe present invention.

FIGS. 5A-5C are examples of video frames illustrating stale backgroundpixels identification, according to one embodiment of the presentinvention.

FIGS. 6A-6C are examples of video frames illustrating gradualillumination problems, according to one embodiment of the presentinvention.

DETAILED DESCRIPTION

Video analysis systems analyze information acquired from observations ofan environment made over time. In context of the present invention,information from a video stream (i.e., a sequence of individual videoframes) is analyzed. In particular, this disclosure describes techniquesfor identifying a background/foreground of a scene depicted in the videostream. Further, embodiments of the invention may be used to analyzeinformation captured in the video stream and to identify and updatebackground and foreground images of the scene. In one embodiment,content of the video stream is analyzed frame-by-frame where each frameis represented as a two-dimensional array of pixel color values. Thebackground image depicts stationary elements of the scene built-up overa sequence of frames, while the foreground image includes volatileelements of the scene built-up over a sequence of frames. That is, thebackground image provides a stage upon which foreground elements mayenter, interact with one another, and leave.

In the following, reference is made to embodiments of the invention.However, it should be understood that the invention is not limited toany specifically described embodiment. Instead, any combination of thefollowing features and elements, whether related to differentembodiments or not, is contemplated to implement and practice theinvention. Furthermore, in various embodiments the invention providesnumerous advantages over the prior art. However, although embodiments ofthe invention may achieve advantages over other possible solutionsand/or over the prior art, whether or not a particular advantage isachieved by a given embodiment is not limiting of the invention. Thus,the following aspects, features, embodiments and advantages are merelyillustrative and are not considered elements or limitations of theappended claims except where explicitly recited in a claim(s). Likewise,reference to “the invention” shall not be construed as a generalizationof any inventive subject matter disclosed herein and shall not beconsidered to be an element or limitation of the appended claims exceptwhere explicitly recited in a claim(s).

One embodiment of the invention is implemented as a program product foruse with a computer system. The program(s) of the program productdefines functions of the embodiments (including the methods describedherein) and can be contained on a variety of computer-readable storagemedia. Illustrative computer-readable storage media include, but are notlimited to: (i) non-writable storage media (e.g., read-only memorydevices within a computer such as CD-ROM disks readable by a CD-ROMdrive) on which information is permanently stored; (ii) writable storagemedia (e.g., floppy disks within a diskette drive or hard-disk drive) onwhich alterable information is stored. Such computer-readable storagemedia, when carrying computer-readable instructions that direct thefunctions of the present invention, are embodiments of the presentinvention. Other media include communications media through whichinformation is conveyed to a computer, such as through a computer ortelephone network, including wireless communications networks.

In general, the routines executed to implement the embodiments of theinvention may be part of an operating system or a specific application,component, program, module, object, or sequence of instructions. Thecomputer program of the present invention is comprised typically of amultitude of instructions that will be translated by the native computerinto a machine-readable format and hence executable instructions. Also,programs are comprised of variables and data structures that eitherreside locally to the program or are found in memory or on storagedevices. In addition, various programs described herein may beidentified based upon the application for which they are implemented ina specific embodiment of the invention. However, it should beappreciated that any particular program nomenclature that follows isused merely for convenience, and thus the invention should not belimited to use solely in any specific application identified and/orimplied by such nomenclature.

FIG. 1 is a block diagram illustrating components of abehavior-recognition system, according to one embodiment of the presentinvention. As shown, the behavior-recognition system 100 includes avideo input 112, a network 114, a computer system 116, and input andoutput devices 118 (e.g., a monitor, a keyboard, a mouse, a printer, andthe like).

The network 114 receives video data (e.g., video stream(s), videoimages, or the like) from the video input 112. The video input 112 maybe a video camera, a VCR, DVR, DVD, computer, web-cam device, or thelike. For example, the video input 112 may be a stationary video cameraaimed at a certain area (e.g., a subway station, a parking lot, abuilding entry/exit, etc.), which continuously records the area andevents taking place therein. Generally, the area visible to the camerais referred to as the “scene.” The video input 112 may be configured torecord the scene as a sequence of individual video frames at a specifiedframe-rate (e.g., 24 frames per second), where each frame includes afixed number of pixels (e.g., 320×240). Each pixel of each frame mayspecify a color value (e.g., an RGB value). Further, the video streammay be formatted using known such formats e.g., MPEG2, MJPEG, MPEG4,H.263, H.264, and the like. The behavior-recognition system 100 may beconfigured to analyze this raw information to identify active objects inthe stream, classify such elements, derive a variety of metadataregarding the actions and interactions of such elements and supply thisinformation to a machine learning engine 134. In turn, the machinelearning engine 134 may be configured to evaluate, observe, learn andremember what events transpire within the scene over time. Further,based on the “learning,” the machine learning engine 134 may identifycertain behaviors as anomalous.

The network 114 may be used to transmit the video data recorded by thevideo input 112 to the computer system 116. In one embodiment, thenetwork 114 transmits the received stream of video frames to thecomputer system 116. Illustratively, the computer system 116 includes aCPU 122, storage 124 (e.g., a disk drive, optical disk drive, floppydisk drive, and the like), memory 126 containing a computer visionengine 132, and the machine learning engine 134. The computer visionengine 132 may provide a software application configured to analyze asequence of video frames provided by the video input 112. For example,in one embodiment, the computer vision engine 132 may be configured toanalyze video frames to identify targets of interest, track thosetargets of interest, infer properties about the targets of interest,classify them by categories, and tag the observed data. In oneembodiment, the computer vision engine 132 generates a list ofattributes (such as texture, color, and the like) of the classifiedobjects of interest and provides the list to the machine learning engine134. Additionally, the computer vision engine may supply the machinelearning engine 134 with a variety of information about each trackedobject within a scene (e.g., kinematic data, depth data, color, data,appearance data, etc.).

Further, as described in greater detail below, the computer visionengine 132 may process video frame data to distinguish betweenbackground elements of the scene and foreground elements of the scene.The background image may represent a static image of the scene, absentany foreground elements, built-up over a sequence of frames. Furtherstill, in one embodiment, the computer vision engine 132 may beconfigured to identify different regions, or segments, of the backgroundimage and identify contextual information about each segment, e.g.,whether one segment is in front of (or behind) another segment. Fromthis information, the computer vision engine 132 may determinedimensionality and geometry of both background and foreground elements.For example, assume a computer vision engine has identified a blob ofpixels as depicting a human individual. And further, that the blob is 30pixels in height. This provides a convenient mechanism for estimatingthe size of objects in the scene, based on the average height of aperson.

In one embodiment, the computer engine 132 may include abackground-foreground module 142 implementing methodologies of thepresent disclosure. Generally, the background-foreground module 142 isconfigured to analyze video frames to identify and/or update sets ofbackground and foreground images and/or background and foregroundsmodels for use by other components of the behavior-recognition system.Generally, the background model specifies how the background-foregroundmodule evaluates scene imagery to compute and update a background image.Typically, the background image is defined as a two-dimensional array ofpixel values specifying a color (or grayscale) value for the backgrounddepicted in the scene imagery. In one embodiment, thebackground-foreground module 142 uses pixel colorcharacteristics/features/attributes of each video frame to identifywhich portions of a given frame depict part of the background image andwhich depict part of a foreground object, according to the backgroundmodel.

Generally, the background image includes stationary elements of thescene being captured by the video input (e.g., pixels depicting aplatform of a subway station), while the foreground image includesvolatile elements captured by the video input (e.g., pixels depicting aman moving around the platform).

Typically, pixels that do not significantly change color over time areconsidered part of the background image of a scene while the rest of thepixels are considered to form foreground objects of the scene. Thus, inone embodiment, a background model may be generated by determining acolor value for each pixel in a background image, where the model is“trained” using a sequence of video frames. For example, one approachincludes determining an average of color values for a pixel over manyframes and using the resulting as the background image. Of course,depending on the anticipated motion (and rate of motion) of foregroundelements within the scene, more sophisticated approaches may be used.Once the background image is “trained,” substantial changes in a colorfor a given pixel from a value “near” the background color to anothercolor value may be interpreted as the appearance of a foreground objectin the scene. Stated differently, for any given frame, each pixel with acolor substantially the same as the background color of that pixel isclassified as depicting a portion of the background image. Motions ofthe foreground objects may be determined based on differences betweenpixel color values in successive video frames. Therefore, the backgroundimage may be envisioned as a video frame of pixels having any foregroundobjects removed from the scene. And conversely, foreground images may beenvisioned as pixels that occlude the background image or,alternatively, as a transparent video frame with patches of theforeground pixels.

In one embodiment, the machine learning engine 134 receives the videoframes and the results generated by the computer vision engine 132. Themachine learning engine 134 may be configured to analyze the receiveddata, build semantic representations of events depicted in the videoframes, detect patterns, and, ultimately, to learn from these observedpatterns to identify normal and/or abnormal events. Data describingwhether a normal/abnormal behavior/event has been determined and/or whatsuch behavior/event is may be provided to output devices 118 to issuealerts, for example, an alert message presented on a GUI interfacescreen.

In general, the computer vision engine 132 and the machine learningengine 134 both process the received video data in real-time. However,time scales for processing information by the computer vision engine 132and the machine learning engine 134 may differ. For example, in oneembodiment, the computer vision engine 132 processes the received videodata frame-by-frame, while the machine learning engine processes 134 thereceived data every N-frames. In other words, while the computer visionengine 132 analyzes each frame in real-time to derive a set ofinformation about what is occurring within a given frame, the machinelearning engine 134 is not constrained by the real-time frame rate ofthe video input.

Note, however, FIG. 1 illustrates merely one possible arrangement of thebehavior-recognition system 100. For example, while the video input 112is shown connected to the computer system 116 via the network 114, thenetwork 114 is not always present or needed (e.g., the video input 112may be directly connected to the computer system 116). Further, variouscomponents and modules of the behavior-recognition system may beimplemented into other systems. For example, in one embodiment, thecomputer vision engine 132 may be implemented as a part of a video inputdevice (e.g., as a firmware component wired directly into a videocamera). In such a case, the outputs of the video camera may be providedto the machine learning engine 134 for analysis.

Moreover, while the background-foreground module 142 is depicted as apart of the computer vision engine 132, it may be implemented as aseparate module placed into the memory 126. The background-foregroundmodule 142 may also be implemented as a system separate from thecomputer system 116 or, alternatively, as a part of a different system.In such a case, the background-foreground module 142 may communicatewith other components of the behavior-recognition system 100 via, forexample, network 114. Furthermore, the methodologies described in thepresent disclosure, including the background-foreground module 142, maybe implemented into any suitable video content analysis system toprovide detailed information about particular objects in a scene.

FIG. 2 illustrates a background-foreground module 200 of a videoanalysis system as well as the functionality and information flow of thebackground-foreground module 200, according to one embodiment of thepresent invention. As shown, the background-foreground module 200includes a supervisor 210, a dark scene compensation module 220, abackground/foreground (BG/FG) classification module 230, a gradualillumination compensation module 240, a stale background/foreground(BG/FG) module 250, a sudden lighting change module 260 and apan-tilt-zoom (PTZ) change module 270.

A video input, such as a sequence of video frames, is supplied to thesupervisor 210. In one embodiment, the supervisor 210 may be configuredto provide the received video frames to the BG/FG classification module230 directly (i.e., provide the raw frame) or provide a frame generatedby dark scene compensation module 220. Generally, if the supervisor 210determines that a scene is too dark to provide an accuratebackground/foreground classification for some region of the raw videoframe (or even the entire frame), the supervisor 210 may pass thereceived video frames to the dark scene compensation module 220.

Generally, when a scene is dark, the amount of light reaching sensors ofa video acquisition device, such as a video camera, is low, and thus,the signal is noisy. Consequently, pixels of video frames depicting adark scene are unstable, i.e., the color value of a given pixel mayfluctuate widely between frames. For example, in one embodiment, basedon the Red-Green-Blue (RGB) color model where the RGB values range from0 to 255, the scene is dark if least one of average RGB values forpixels of a video frame is lower than 50. Due to low RGB values, evensmall changes in the RGB values have significant effect inclassification of pixels and may result in pixels being misclassified,such as classifying foreground pixels as background pixels and viceversa. For example, when an R channel value for a pixel is 4 in onevideo frame and 6 in another video frame, the difference between the twois only 2. However, because the R channel value is so low to begin with,the difference of 2 actually represents a fifty percent variance. Inother words, because the video signal is weak, the noise swallows thevideo signal.

To address this issue, in one embodiment, the dark scene compensationmodule 220 smoothes such video signals, reducing the noise. To reducethe noise, the dark scene compensation module 220 calculates averagecolor-characteristic values (such as RGB values) over a sequence ofconsecutive frames (e.g., 5 frames) and assigns the resulting averagecolor-characteristic values to pixels of a current video frame. That is,when the scene (or region of the scene) is dark, the compensation modulemay substitute actual pixel color in the raw frame with an averagedetermined from previous values of that pixel. In this manner, eachvideo frame provided to other components of the background-foregroundmodule 200 from the dark scene compensation module 220 is a rollingaverage of N preceding video frames.

The number of frames chosen to average the color-characteristic valuesmay differ between different embodiments. Generally, using more framesin calculating average color-characteristic values, e.g. 10 framesinstead of 2-5, allows for more accurate reflection by such values ofobjects depicted in the scene. On the other hand, using too many framesmay cause blurring of the foreground, especially fast moving foregroundobjects, into the background. In one embodiment, the number of framesused to calculate average color-characteristic values depends on theobserved environment, e.g., whether the environment is a slow changingor rapidly changing environment. Further yet, in another embodiment, thenumber of frames used to calculate average color-characteristic valuesmay vary depending on a current level of darkness in the scene.

Note, however, that assigning average color-characteristic values asvalues of the currently received frame does not disrupt functionality ofthe other components of the background-foreground module 200, such asBG/FG classification module 230. In one embodiment, the other componentsof the background-foreground module 200 are simply unaware that a videoframe they receive has been altered by the dark scene compensationmodule 220 to reduce noise of the video signal. Instead, running thevideo frames through the dark scene module 220 results in a short delayin identifying background and/or foreground information, similar to adelay in processing information by a person before his or her eyesadjust to darkness.

In contrast, there is no delay in bypassing the dark scene module whenthe supervisor 210 determines that the scene is not dark anymore, assuch a determination may be made over one frame. Whether a scene issufficiently dark may be determined by comparing color-characteristicvalues to a pre-defined threshold, such as comparing the RGB values to apre-defined value (e.g., R channel value below 40). It should be notedthat such a threshold may vary between different embodiments and may betailored to the specific needs in an individual case, e.g., the type ofenvironment depicted in the video frames. Furthermore differentthresholds may be used for different color-characteristics of pixels,such as for different RGB channels, RGB values of different range (e.g.,0-255 vs. 0-65535), RGB values vs. YCbCr values, and so on.

Generally, the BG/FG classification module 230 analyzes pixels of acurrent video frame and classifies them as depicting a portion of scenebackground or a foreground element. Also, the BG/FG classificationmodule 230 supports one or more background models to facilitate pixelclassifications. In such a case, different background models eachcompute an independent a background image for a given scene. Asdiscussed previously, the background of the scene typically includesstationary elements of the scene, while the foreground of the sceneincludes volatile elements of the scene. Accordingly, changes in pixelcharacteristics over time (e.g., over a sequence of frames) are used todetermine the background and foreground of the scene.

In one embodiment, a background model includes the same number of pixelshaving the same coordinates as the video frames. While a given pixel maydepict background of the scene in one video frame and foreground of thescene in another video frame, the background model stores a set of pixelcolor-characteristics for each pixel representing the background of thescene. Further, pixels with color-characteristic values that do notsignificantly change over time may also be classified as backgroundpixels. For example, after a foreground object appears within the sceneit may become stationary (e.g., a car parks in a parking spot). Inresponse, after a period of time, the set of foreground pixels depictingthe foreground object may eventually dissolve into the background.Non-background pixels form a foreground of the scene.

Whether a given pixel is classified as depicting background orforeground of the scene may be determined for each frame in a sequenceof video frames. For example, foreground elements may be identified bycomparing color-characteristic values for a given pixel with thecolor-characteristic values of a corresponding pixel of the backgroundimage of the scene. Further, as video frames are received, thebackground image may be updated using additional information fromsuccessive frames. Such updates may improve the accuracy of thebackground image of the scene and allow the background model to adapt tochanges in the scene.

Whether a particular pixel of the current video frame is classified as abackground pixel may be determined using a comparison measure of thatpixel. In one embodiment, the comparison measure is determined based ondifferences identified using color-characteristics of the pixel andcolor-characteristics of a corresponding pixel in the background image,where such differences are considered in light of a range of possiblevalues of the color-characteristics. The pixel comparison measure may becompared to a pre-defined threshold to classify the pixel as abackground or foreground pixel. In one embodiment, the pre-definedthreshold depends on a type (e.g., bright, dark, and so on) assigned tothe pixel. That is, the threshold applied to given comparison measurecalculated for a pixel may be different, depending on attributes of thepixel. The comparison measure and its calculation and use are describedin greater detail below, in conjunction with a discussion of FIG. 4.

As shown in FIG. 2, the BG/FG classification module may include ashort-term background component (STBG) 232 and a long-term backgroundcomponent (LTBG) 234. Generally, these components may be used tomaintain multiple background models of the scene. For example, in oneembodiment, two background models are used—a short-term background modeland a long-term background model. Typically, the short-term backgroundmodel is updated more rapidly than the long-term background model. Forexample, when a new object appears in the scene (e.g., a floor lampplaced in a stationary position), the foreground object does notimmediately become a part of either the short-term background model, orthe long-term background model. Instead, pixels depicting the object areclassified as being part of the foreground, i.e. as a foreground object.However, as time passes (e.g., video frames are received and analyzed)and the foreground object remains motionless, it becomes a part of theshort-term background. In contrast, the long-term background may notincorporate pixels forming the foreground object for a longer period oftime, or not at all. In other words, the short-term background modelincorporates new objects in the scene into the background more readilythan the long-term background model. Conversely, the long-termbackground model is more resistant to change.

The amount of time (or number of video frames) needed before a pixelclassified as depicting a foreground object becomes classifieds asshort-term and/or long-term background varies between embodiments andmay depend on, for example, the nature of the observed environment. Forexample, for an outdoor scene, where the background changes rapidly(shading, movements of tree branches, and so on), a background model mayneed to be more adaptable than for an indoor scene, where the backgroundmay be more consistent. In one embodiment, it takes several framesbefore a pixel exhibiting consistent values becomes a part of theshort-term background model. Therefore, for example, if a long car, suchas a limousine, moves slowly through a parking lot, the middle part ofthe car may become a part of the short-term background model, but not ofthe long-term model. In another embodiment, once a foreground object hasbeen identified as such, pixels classified as depicting that object donot become a part of the long-term background model while exhibitingcolor-characteristic values consistent with the identified foregroundobject.

Note however, results of the analysis made by the BG/FG classificationmodule 230, including the classification of a pixel as depictingbackground or foreground and/or updates made to the short-term andlong-term background models may be overridden via an outside input, forexample, via a human interference. Alternatively, other components ofthe video-analysis system or behavior-recognition system may overridethe results of the BG/FG classification module 230. In one embodiment,components of computer vision engine 132, such as a tracker, contextprocessor, and estimator-identifier, may re-classify an object (asrepresented by a collection of pixels) as depicting part of thebackground of the scene after this object has been classified as aforeground object by BG/FG classification module 230. For example,assume that a recorded scene depicts a parking lot having severalparking spots. Assume further, that at some point in time a handicapparking sign is painted on one of the parking spot. Because thecolor-characteristic values of pixels depicting this parking sign havechanged, the BG/FG classification module 230 may classify thehandicapped parking sign as a foreground object. However, for example,when the tracker identifies that the handicapped sign is of no interestand should not be tracked, the tracker may signal to the BG/FGclassification module 230 that the pixels depicting the handicappedparking sign should be classified as part of the background.

As shown in FIG. 2, the background-foreground module 200 may alsoinclude an illumination compensation module 240. In one embodiment, theillumination compensation module 240 may be configured to reduce thepossibility of pixel misclassifications caused by lighting changessweeping across a scene, such as changes caused by shadows. For example,as a shadow moves over a grassy field, the pixels at the leading edge ofthe shadow may be classified as a foreground object due to changes incolor value, even though the pixels depict the background of the scene.Typically, pixels within a shadow do not become a part of the backgroundmodel because when the shadow moves over the portion of the scenecaptured by a given pixel, the pixel values change with sufficientmagnitude to be classified as being foreground.

To address the issue of pixel misclassification caused by lightingchanges sweeping across a scene, the illumination compensation module240 may be configured to perform a second evaluation of a pixelclassified as being part of the foreground when the pixel is classifiedas being part of the foreground by a small amount. That is, when thepixel is within the threshold for being classified as foreground, butonly within a marginal difference between the threshold and the pixelcolor values. How small before the magnitude of such a difference isconsidered “small” may be tailored to suit the needs of particular case.Further, a map of possible lighting compensation problems may begenerated by calculating the difference between the threshold forbackground/foreground classification and the pixel values for each pixelin a given frame. Doing so creates a map where the leading (andtrailing) edge of sweeping lighting changes (such as those caused by ashadow) may readily be identified. In such a case, a bias may becalculated that may push the classification of the pixel towards (oraway from) the threshold as the leading edge of the shadow (or othersweeping lighting changes) crosses over a given pixel.

In one embodiment, the bias for given pixel values is calculated as anaverage pixel color value of pixels neighboring the given pixel. Forexample, consider a pixel just struck by the leading edge of a shadowand, as a result is classified as depicting a foreground object of thescene. In such a case, a number of pixels “ahead” of the leading edge ofthe shadow may still depict background using a brighter set of colorvalues and another set of pixels “behind” the leading edge of the shadowmay depict the background with a darker set of color values. By takingan average of these pixels (e.g., a grid of 9 pixels, with the givenpixel in the center), the pixel just struck by the leading edge of theshadow may be reclassified as depicting scene background if the averagecolor values are within the threshold for being classified asbackground. In this manner, the gradual illumination compensation module240 helps to prevent misclassification of pixels that could have beencaused by a leading edge of a rolling shadow (or other gradual lightingchanges).

It should be noted that not every embodiment of the present inventionapplies the gradual illumination compensation module 240 while analyzingthe stream of frames or even includes this module. For example, in oneembodiment, where the observed environment is susceptible to causing alot of marginal classifications (pixel comparison measures are close topre-defined thresholds), it may be beneficial to omit using the gradualillumination compensation module 240 to avoid over re-classification.Alternatively, the thresholds or bias may be adjusted to improvebackground/foreground classifications.

In one embodiment, the stale BG/FG module 250 may be configured toidentify stale background pixels, and in response, mature the backgroundimage when stale background pixels are discovered. A background pixel isconsidered “stale” when the color value associated with a given pixelbecomes outdated from a current color of the background in the scene.This may result in a pixel that should be classified as depictingbackground in a given frame to become classified as a foreground pixel.Consider, for example, the following scenario. Assume a parking lotbeing monitored via video. In such a case, the empty parking lot wouldbe classified as background and incorporated into a background image.That is, the color value of each pixel depicting the empty parking lotwould become the color value assigned to a corresponding pixel in thebackground image. Assume further, that a car appears and parks in one ofthe parking spaces for a period of time. While parked, the car occludesa part of the background. Assume further that by the time the carleaves, environmental conditions of the parking lot have changed, e.g.,the sun has set, and thus, it becomes darker. In such a case, the colorvalues of regions of the parking lot not obstructed by the car may havebeen updated to reflect the gradual change in lighting. At the sametime, however, the color-characteristic values of background imagepixels depicting the part of the parking lot occluded by the car werenot. Consequently, the color values of these pixels are the same as atthe time before the car appeared in the parking lot. These pixels arestale.

As these stale pixels have not been updated, the corresponding pixels ofthe current video frame may be misclassified as foreground pixels,although these pixels depict the background of the scene, i.e., an emptyparking space. Such misclassification occurs because of a mismatchbetween color values of the background image and color values of abackground depicted in the current video frame. Another example alongthe same lines includes that of a car parked in a spot when thebackground image was trained. So long as the car remains stationery, itis properly classified as part of the background. However, once the carleaves, the region “uncovered” by the departure of the car ends up beingclassified as part of the foreground of the scene. In this latterexample, a portion of the background has simply “left” the scene.

To avoid, or at least to reduce, misclassifications resulting from“stale” background pixels, the stale BG/FG module 250 may be configuredto evaluates background pixels neighboring the stale background pixelsand compares their color-characteristic changes from the time theforeground object appeared in the scene (e.g., the car parked) to thetime the foreground object left the scene (e.g., the car left theparking spot) with the respective color-characteristic changes of thestale background pixels. If the changes are sufficiently similar, thenthe stale BG/FG module 250 modifies the background image by updating thecolor-characteristic values of the stale pixels to reflect the values ofthe current video frame. In one embodiment, this similarity may bedetermined by comparing differences between Euclidean distancesrepresenting changes to the color values of unobstructed pixels and theobstructed pixels to a pre-defined threshold. Aspects of the stale BG/FGmodule 250 are further described below in conjunction with FIGS. 5A-5C.

The sudden lighting change module 260 is generally responsible foridentifying drastic changes in lighting of the scene. A drastic changein lighting may be caused, for example, by turning a light on and off.When lighting suddenly changes, the color-characteristic values changedrastically as well (e.g., over a small sequence of frames, typically4-5). Accordingly, the background image, as it was before the suddenlighting change, becomes practically useless for at least a period oftime because values of the color-characteristics of pixels in thecurrent video frame have changed, whether those pixels are foregroundpixels or background pixels. In other words, the background pixels ofthe current video frame cannot be identified as such because they nolonger match the pixels of the background image. In one embodiment, whena sudden lighting change occurs, the background image is reset andre-trained over a series of frames, starting with the frame in which thesudden lighting change occurred.

In one embodiment, the number of pixels classified as depicting sceneforeground in a current video frame is compared with the number ofpixels classified as depicting scene background in order to determine asudden lighting change. Finding a drastic change between the current andprevious foregrounds may indicate that a sudden lighting change hasoccurred. Thus, a foreground change between the current and previousforegrounds may be evaluated, for example, by calculating the differencebetween a total number of the foreground pixels of the current videoframe and a total number of foreground pixels of the previous videoframe. Alternatively, the foreground change is determined by calculatinga number of background pixels of the previous video frame that becameforeground pixels of the current video frame and comparing the number toa pre-defined threshold. For example, if the pixels classified as theforeground pixels of the current frame, but not as the foreground pixelsof the previous frame, represent more than eighty percent of allforeground pixels of the current video frame, the foreground change isconsidered drastic.

Further, pixels depicting corners of objects in the current video framemay be analyzed to confirm whether a sudden lighting change hasoccurred. For example, if the color values of pixels depicting objectedges or intersections between objects exhibit similar changes betweentwo frames, then a sudden lighting change may be confirmed. In oneembodiment, for each corner pixel a difference value is determined bycalculating the difference between the corner pixel'scolor-characteristic values in the current video frame and the previousvideo frame. Then, the calculated difference values for all cornerpixels are compared and if they are similar (e.g., do not differ morethan a pre-defined threshold) the sudden lighting change is confirmed.Alternatively, the spatial-temporal neighborhood can be analyzed tocheck if the intensity changes are consistent with the effect of suddenillumination changes in that area.

The PTZ changes module 270 determines whether there is any pan, tilt, orzoom change in the scene indicating that the video acquisition devicehas been moved. In one embodiment, if such a change is determined, thebackground image is reset and other components of thebehavior-recognition system are informed about the detected change sothey may also reset their learning models.

Note, however, FIG. 2 illustrates merely one possible arrangement of thebackground-foreground module 200. Not all illustrated components arealways present and/or needed. Furthermore, various components of thebackground-foreground module 200 may be implemented into the othercomponents or combined with other components. For example, while thesupervisor 210 as depicted only decides whether the dark scenecompensation module 220 should be employed, in another embodiment, thesupervisor 210 may also decide when and whether the other components(such as gradual illumination compensation module 240) of thebackground-foreground module 200 should be used in the video contentanalysis. Moreover, although the above described analysis of the videoimages were primarily explained on pixel-by-pixel basis, it is merelymatter of convenience. The principles of the present invention may besimilarly applied to blocks or groups of pixels.

FIGS. 3A-3B are flow diagrams illustrating a method 300 for identifyinga background of a scene depicted in a stream of video frames, accordingto one embodiment of the invention. As shown, the method 300 begins atstep 302. At step 304, a current video frame (i.e., raw video frame) isreceived. At step 306, it is determined whether a background (BG) modelof the scene has been trained. Typically, at the beginning of videoanalysis, the background model has not been trained yet. That is, thebackground model does not initially provide a model of the background ofthe scene. Also, in some circumstances, the background model may bereset and re-trained. In either case, if the background model has notbeen trained the method 300 continues with a step 308, where thebackground model is trained.

Generally, the background model is used to assign color-characteristicvalues determined by accumulating color-characteristic values (such asRGB values) of the pixels over a sequence of video frames (e.g., a fewhundred frames). For example, a simple approach may be to simplycalculate an average for each pixel from the values in each of thesequence of frames. However, a more sophisticated approach may filterframes determined to include outliers for some pixel values. As shown,the training includes steps 308, 304 and 306, which repeat until thebackground model is trained. As another alternative, an initialbackground image may be provided (e.g., a frame captured prior to anyactivity occurring within the scene). In such a case, no training isnecessary. Of course, these, and other, approaches may be used todetermine an initial background image for a given scene.

At step 310, the background foreground model 250 may determine whetherthe scene depicted in the current video frame is “dark.” Generally, thismay be determined by comparing color-characteristic values topre-defined thresholds. Of course, the thresholds required to determinethat the scene is dark may be tailored to suit the needs of a particularcase. In any event, if at least one of the color-channels is below thepre-defined threshold, the supervisor module may determine that thescene is dark and the method 300 continues with step 312. Otherwise, themethod 300 continues with step 318.

As discussed above, pixels forming a dark scene are typically unstablefrom frame-to-frame. To address this issue, at steps 312, 314, and 316,the color-characteristic values of the pixels are accumulated over asequence of video frames and averages are calculated. As shown in FIG.3, at step 316, it is determined whether the averages compensate forscene darkness. If the pixels have not been stabilized, an additionalvideo frame is received (step 312) and the average color-characteristicvalues are calculated with the pixel characteristics of the additionalframe (step 314). Otherwise, a video frame that includes pixels valuescalculated as an average of multiple frames is provided to thebackground foreground model as a current video frame.

In one embodiment, the dark scene determination is not necessary donefor the whole video frame. Rather, dark regions of the current videoframe are determined. In such an embodiment, the above-described steps312, 314, and 316 are applied only to the pixels of the dark regions.Accordingly, a video frame provided after the completion of steps 312,314, and 316 includes pixels of the dark regions having averagecolor-characteristic values assigned as their current values and pixels,located outside of the dark regions having color-characteristic valuesof the raw video frame.

As discussed above in greater detail, a sudden lighting change may causethe misclassification of the background pixels as the foreground (e.g.,lights turned on or off). To avoid such a misclassification, at step318, it is determined whether the lighting of the scene has drasticallychanged. In one embodiment, a sudden increase in the amount of pixelsclassified as foreground pixels between the current and previous videoframes indicate a sudden lighting change of the scene. Such a suddenlighting change may be further confirmed by evaluating whether cornerpixels of the video frame exhibit similar changes (e.g., a Euclidiandistance) indicating an overall change in scene lighting for the objectsdepicted in the scene. When a sudden lighting change occurs, thebackground model is reset (step 320), and the method 300 returns to step304. Otherwise, the method 300 continues with step 322.

At step 322, it is determined whether a comparison measure has beencalculated for each pixel of the current video frame. If so, the method300 proceeds to step 344. Otherwise, at step 324, the comparison valuefor pixels of the current video frame is determined. As described below,the comparison measure may be determined based on a calculateddifference between color-characteristics of a pixel and a correspondingpixel in the background model. In one embodiment, the calculateddifferences may be evaluated relative to a range of possiblecolor-characteristic values. After the comparison measure has beendetermined, its value is compared to a threshold to classify the pixelof the current video frame as being part of either the background orforeground of the scene (step 324). Specifically, if the comparisonvalue is above the threshold, the pixel is classified as the foregroundof the scene (step 326). Otherwise, the pixel is classified as thebackground of the scene (step 340).

At step 330, the classification of a given pixel as depicting backgroundor foreground may be evaluated to avoid potential misclassification dueto by regional lighting changes (discussed above in regard to thegradual illumination compensation module 240). Specifically, at step330, the pixel is analyzed to determine whether its color-characteristicvalues exhibit significant changes. For example, when a given pixel isonly marginally classified as depicting scene foreground, an average setof color values may be calculated using the color values of a set ofpixels neighboring the given pixel (step 332). If the calculated averagevalue within the threshold for being classified as scene background,then the given pixel may itself be reclassified as scene background(step 338). This approach allows the leading (and trailing) edge of ashadow to sweep across a scene without causing pixels to bemisclassified as being part of scene foreground.

If no re-classification occurs, then at step 334, it is determinedwhether the pixel represents a stale background pixel. Step 334 helpsprevent pixels from being misclassified as part of the foreground whenthey, in fact, depict the background of the scene. As previouslydiscussed, stale background pixels are pixels of the background imagethat have outdated color-characteristic values, relative to the actualbackground depicted in the scene. For example, if conditions of thescene change while a foreground object occludes the background, then,when the foreground object leaves the scene, the pixels may remainclassified as foreground, because their color values do not match thecolor values of the existing background image.

To assure proper classification in such a scenario, at step 334, pixelsof the background image neighboring a pixel corresponding to the pixelin question are located. The color-characteristic value changes of theneighboring pixels from the time of the last update of the correspondingpixel to the current point in time are determined. If the changes aresufficiently similar to the color-characteristic value changes of thepixel in question over the same period of time, then the correspondingpixel of the background image is a stale pixel. Accordingly, at step336, the pixel of the current frame is re-classified as a backgroundpixel and its color-characteristic values are used to update thecolor-characteristic values of the stale pixel of the background imageand the method 300 proceeds to step 342. Otherwise, the method 300returns to step 322.

At step 342, differences between the color-characteristic values of thepixel and of the corresponding pixel of the background image aredetermined. The calculated differences are stored in the compensationlighting problems map, which, as discussed, may be used at step 330. Themethod 300 returns to step 322 to evaluate additional pixels of thecurrent video frame, according to steps 324-342.

Once the comparison measure has been calculated for each pixel of thecurrent video frame, the background image of the scene may be updated(step 344) using information determined during the above-describedanalysis of the current video frame. After the background model has beenupdated, the method 300 returns to step 304 to receive the next rawvideo frame.

Although steps 322 through 342 are described as being performed on apixel-by-pixel basis, in one embodiment, these steps are performed onblock-by-block basis. Furthermore, it is not necessary to perform all ofthe above-described steps in the order named. Moreover, not all of thedescribed steps are necessary for the described method to operate. Whichsteps should be used, in what order the steps should be performed, andwhether some steps should be repeated more often than other steps isdetermined, based on, for example, the needs of a particular user,specific qualities of an observed environment, and so on.

For example, in one embodiment the background model is updated onlyevery K frames. Alternatively, steps 330, 332, 338 and 342 may beomitted in some cases, e.g. where the scene depicts an indoorenvironment where gradual lighting changes are unlikely to occur.Similarly, not every described step need be performed on apixel-by-pixel basis.

FIG. 4 is a flowchart illustrating a method 400 for using a comparisonmeasure to identify a background of a scene, according to one embodimentof the invention. The method 400 of FIG. 4 uses a pixel-based examplefor using a comparison measure to build a background image of a scene.Of course, other approaches may be used. For example, another approachto a background model includes a flow based approach which identifiesmoving (and non-moving) objects in the scene, and assigns the colorvalues of the non-moving objects to the background image. The method 400starts with step 405. At step 410, it is determined whether each pixelof a current video frame has been classified as being a background orforeground pixel. If so, the method 400 ends with step 455. Otherwise,if the current video frame still has unclassified pixels, the method 400continues with step 415.

At step 415, a pixel of a current video frame is chosen forclassification. For each color-characteristic of the pixel, a differencebetween color values of the pixel and color values of a correspondingpixel in the current background image. Generally, each pixel of thebackground image corresponds to a pixel of the current video framesharing the same pixel coordinates. To facilitate this description ofthe method 400 the following example may be helpful. In this example,the color-characteristics of a pixel are defined using R, G, and Bchannels. Table 1 represents RGB values of one pixel of a current videoframe and a corresponding pixel of a background image.

TABLE 1 RGB channels R G B RGB values of a pixel of the current videoframe - 115 89 90 Rcf, Gcf, Bcf RGB values of a corresponding pixel ofthe background 129 72 6 image - Rbg, Gbg, Bbg Absolute differences ofthe RGB values - |Rbg − Rcf|; 14 17 74 |Gbg − Gcf|; | Bbg − Bcf|

As shown in Table 1, values of the R, G and B channels of a pixel of thecurrent video frame are 115, 86 and 90. Values of the R, G and Bchannels of the corresponding pixel of the background image are 129, 72,and 6. Accordingly, the differences in these channels (ΔR, ΔG and ΔB),may be calculated based on differences between values of the RGBchannels of the pixel (namely, Rcf, Gcf or Bcf) and RGB values of thechannels of the corresponding pixel (namely, Rbg, Gbg or Bbg). In oneembodiment, the channel-differences ΔR, ΔG and ΔB are calculated usingthe following equations:

$\begin{matrix}{\Delta_{R} = \frac{255}{{{R_{bg} - R_{cf}}} + 1}} & (1) \\{\Delta_{G} = \frac{255}{{{G_{bg} - G_{cf}}} + 1}} & (2) \\{\Delta_{B} = \frac{255}{{{B_{bg} - B_{cf}}} + 1}} & (3)\end{matrix}$

Applying the equations (1), (2), and (3) to values contained in Table 1,the channel-difference values for the described example are thefollowing: ΔR=17, ΔG=15 and ΔB=3.4.

Note, although the above equations use 255 to represent the range ofpossible values of each of the RGB channels, the present invention isnot limited to this number. For example, if RGB channels are 16-bitchannels, the RGB values may vary from 0 to 65535. Furthermore, in oneembodiment, a range of possible values corresponding to a type of acolor-characteristic employed may be purposefully decreased, forexample, to address specific characteristics of a video acquisitiondevice. Further, the above-described equations are merely examples ofequations used according to the principles of the present invention.

In one embodiment, to reduce video noise effects on pixelclassification, for each pixel of the current video frame, a pixel colorchannel most affected by video noise is determined. Such a channel maybe determined by identifying as the one having a minimum difference inthe calculated difference values of the pixel (step 420). Thus, in theabove-described example, ΔB (which is 3.4 compared to 17 of ΔR and 15 ofΔG) is such a value.

At step 425, a comparison measure for the pixel is determined bycalculating the average of the difference values. Note, however, thevalue identified at step 420 may be omitted from the calculations toreduce video noise effect on pixel classification. Accordingly, in theabove-described example, the comparison measure is calculated as theaverage of ΔR and ΔG (which makes the comparison for this example pixelmeasure equal to 16), because ΔB, having the smallest value, is omittedfrom the calculations.

At step 430, a type of the pixel is determined. For example, brightnessof the pixel may be used to determine its type. Because pixels ofdifferent brightness exhibit different physical properties, differentpixel models may be employed. In one embodiment, the pixel may be one ofthe three types, namely bright (e.g., where at least one of the RGBchannels is above 200), dark (e.g., where at least one of the RGBchannels is below 50), and medium (e.g., the rest of the RGB values). Itshould be noted that more or less than three pixel types may be used.Furthermore, the RGB values provided are merely examples, and other RGBvalues may be used to define boundaries between the pixel types. Forexample, which RGB values are used to define pixel type boundaries maydepend on overall brightness of the observed environment.

At step 435, a threshold for classifying a pixel as background versusforeground s selected based on the determined type of the pixel. Asmentioned above, different types of pixels exhibit different properties.Accordingly, a dynamic threshold may be assigned and used to determinewhether the pixel depicts background or foreground. That is, thethresholds assigned to different pixels may be assigned dynamically andvary based on the content of the video, e.g., based on how rapidlyillumination levels change in a given neighborhood or region of thescene. For example, pixels in regions identified to have a largemagnitude of change (i.e., more noise) from frame-to-frame would have adifferent threshold than pixels in regions depicting a well-lit areawith little change in color values of an object depicted in thebackground from frame-to-frame.

In one embodiment, a volatility mask may be calculated to create a mapof how much volatility is occurring in the pixel color values fromframe-to-frame. The volatility mask may indicate how much pixel colorvalues change frame-to-frame, when a given pixel depicts the same objectwithin the scene (i.e., a color value change not due to the movement orintroduction of an object into the foreground of the scene). Areas ofhigh volatility (i.e., high-noise) have more tolerance to changes beforedeclaring that a pixel depicting background in one frame depicts aforeground object in a subsequent frame. Thus, the thresholds fordeclaring a pixel as depicting background or foreground may adapt overtime to changes in the scene and lighting conditions therein.Accordingly, different threshold values may be used to classify pixelsof the current video frame as being part of background versusforeground. At step 440, the comparison measure of the pixel is comparedto the selected threshold. If the comparison measure is above thethreshold, then, at step 445, the pixel is classified as a backgroundpixel of the current video frame. Otherwise, at step 450, the pixel isclassified as a foreground pixel of the current frame. In either case,the method 400 returns to step 410.

It should be noted that it is not necessary to perform all of theabove-described steps in the order named. Furthermore, not all of thedescribed steps are necessary for the described method to operate. Whichsteps should be used, in what order the steps should be performed, andwhether some steps should be repeated more often than other steps isdetermined, based on, for example, the needs of a particular user,specific qualities of an observed environment, and so on. For example,in one embodiment of the present invention, steps 430 and 435 areomitted. Instead, at step 440 the same threshold value (e.g., a changegreater than 10) is used for every pixel, regardless of underlyingbrightness.

Also note, although method 400 is described on a pixel-by-pixel basis,in one embodiment, the described steps (including the classificationstep) are applied to groups of pixels. Moreover, although theabove-described example uses the RGB values as color-characteristicvalues, the present invention is not limited to the RGB values. Othercolor-characteristic values may be used, for example, YCbCr values.Furthermore, while the RGB values provide for three types of thecolor-characteristics, namely R, G and B channels, the principles ofpresent invention may be applied to color models having more than threecolor-characteristics, for example, the CMYK color model, which is afour color model.

FIGS. 5A-5C illustrate examples of video frames depicting a scene takingplace in a parking lot 500, according to one embodiment of theinvention. Specifically, FIG. 5A illustrates the parking lot 500 at 8:00AM, 5B illustrates the parking lot 500 between 8:01 AM and 8:59 PM, and5C illustrates the parking lot 500 at 9:00 PM. In FIGS. 5A-5C, elements505, 510, 515, and 520 respectively indicate parking spots 1-4 of theparking lot 500. According to the above-discussed principles of thepresent invention, a car 525 and a truck 530 may be considered parts ofthe foreground, while the parking spots 505, 510, 515, and 520 may beconsidered parts of the background (assuming the car 525 and the truck530 appeared after the background model was trained). In othercircumstances, the car 525 could be considered as a part of thebackground and background model. For example, if the background imagehas been trained with the car 525 already present in the scene, the car525 will be part of the background until it leaves the scene. In such acase, the “hole” left when car 525 leaves would be misclassified as aforeground object.

In FIGS. 5A-5C, the parking spots 505 and 515 are empty, while theparking spot 520 is occupied by the car 525. The truck 530 is not shownat 8:00 AM (FIG. 5A). In FIG. 5B, the truck is shown having parked inthe parking spot 510 between 8:01 AM and 8:59 PM (FIG. 5B), and shownhaving left the parking spot 510 at 9:00 PM (FIG. 5C). As the timeelapses, the amount of light present in the scene changes. Thus, whileFIG. 5A illustrates the parking lot 500 when the sun is up, FIG. 5Cillustrates the parking lot 500 when the sun is setting, and thus it issignificantly darker than in FIG. 5A. Consequently, the backgroundpixels, including background pixels representing parking spots, havedifferent color-characteristic values in FIGS. 5A and 5C.

Over time, between 8:00 AM and 9:00 PM the background image of the scenedepicted in FIGS. 5A-5C is adapted to reflect changes incolor-characteristic values. Thus, pixels of the background image havedifferent pixel color values at 8:00 AM (FIG. 5A) than at 9:00 PM (FIG.5C). Such changes take place gradually, as the lighting of the scenechanges. As the background image is updated, the background pixels aregradually updated throughout the day.

Nevertheless, these changes take place only for pixels of the backgroundimage that remain visible to the video acquisition device. Therefore,while values of pixels in the background image that depict parking spots505 and 515 may gradually change between 8:00 AM and 9:00 PM, the colorvalues of the pixels representing an area 535 of the parking spot 515have not changed because the truck 530 obstructs the area 535 duringthis period of time. As a result, when the truck 530 leaves the parkingspot 515, the pixels of the background image representing the area 535retain the color-characteristic values as determined at 8:00 AM, whichare different from the color-characteristic values of the pixelsrepresenting the parking spots 505 and 515. The pixels of the videoframe illustrated in FIG. 5C, which represent the area 535, may beclassified as foreground pixels, because their color-characteristicvalues do not match the background image. The pixels of the backgroundimage representing the area 535 would be classified as stale backgroundpixels.

As discussed above, to address such a problem, the stale BG/FG module250 may be configured to analyze how neighboring areas, such as groupsof pixels representing parking spots 505 and 515, have changed andcompare the changes to the changes of color values of the pixelsrepresenting the area 535. If the stale BG/FG module 250 determines thatthe changes are similar (e.g., distance changes between thecolor-characteristic values for the pixels representing parking spots505 and 515 are similar to distance changes for the color-characteristicvalues of the pixels representing the area 535), then the pixels of thebackground image corresponding to the area 535 may be updated withcurrent values. In this example, the color values of the pixels of thevideo frame illustrated in FIG. 5C, which represent the area 535.

FIGS. 6A-6C illustrate examples of video frames showing gradualillumination problems identified according to one embodiment of theinvention. Specifically, FIGS. 6A-6C show three video frames of anoutdoor scene, where the lighting conditions for one of the pixels ofthe video frames, namely pixel 630, change between the shown frames.Illustratively, FIGS. 6A-6C each includes a sun 605, a tree 610, a treeshadow 615, a cloud 620, a cloud shadow 625 and pixel 630. The cloud 620occupies different pixels in each of the shown video frames, movingtowards the tree 610 between FIG. 6A and FIG. 6C. Similarly, the cloudshadow 625 moves toward the tree 610. It should be noted that there areno foreground objects shown in FIGS. 6A-6C.

As the cloud 620 moves, its cloud shadow 625 approaches the pixel 630 inFIG. 6A, overlaps the pixels 630 in FIG. 6B, and leaves the pixel 630 inFIG. 6C. Accordingly, the pixel 630 has different color-characteristicvalues as between FIGS. 6A and 6B and as between FIGS. 6B and 6C. Thechanges in the color-characteristic values take place just over oneframe, namely when an edge 635 of the cloud shadow 625 first reaches thepixel 630, and then, when the edge 635 leaves the pixel 630 as the cloudshadow 625 leaves the pixel 630. Consequently, the changes aresignificant and may lead to classifying the pixel 630 as the foregroundof the scene.

To reduce chances of such a misclassification, a map of existingcompensation lighting problems is created and maintained. As discussedabove, whenever a pixel of a current video frame is classified as abackground pixel, a difference between the color-characteristic valuesof that pixel and the color-characteristic value of a correspondingpixel of the background image is measured and stored as a part of themap. Accordingly, such a map created and maintained fro the scene ofFIGS. 6A-6C includes each pixel of the depicted scene as there are noforeground objects present.

When the edge 635 of the cloud 625 overlaps the pixel 630 for the firsttime, the pixel 630 may initially be classified as being part of theforeground. Accordingly, pixels neighboring the pixel 630 are evaluatedusing values acquired from the map of compensation lighting problems andthe average change value is calculated to determine whether a pixel ofthe background image corresponding to the pixel 630 is a stalebackground pixel. As each grass pixel, overlapped by the cloud shadow625, exhibits a significant change in its color-characteristic value,the calculated average change value should also exhibit a similarchange. Accordingly, classification of the pixel 630 will be biasedtowards being classified as the background. A similar scenario occurswhen the trailing edge of the cloud 635 together with the cloud 620leaves the pixel 630. In this manner, the pixel 630 is re-classified asthe background pixel despite significant changes in itscolor-characteristic values.

Advantageously, as described herein, embodiments of the invention enableidentifying background and foreground images of a scene depicted by astream of video frames and maintaining the background/foregroundclassification as objects move about the scene over time. Furthermore,embodiments of the invention enable maintaining accuratebackground/foreground classification of the scene, when the scene itsregions are dark and/or when lighting conditions of the scene changeover time or occur only in parts of the scene. Moreover, embodiments ofthe invention enable classifying accurately background pixels that wereoccluded by foreground objects while the conditions of the scene werechanging.

While the foregoing is directed to embodiments of the present invention,other and further embodiments of the invention may be devised withoutdeparting from the basic scope thereof, and the scope thereof isdetermined by the claims that follow.

1. A method for analyzing a scene depicted in a sequence of videoframes, the method comprising: receiving a raw video frame of thesequence of video frames; for each of a plurality of pixels of the rawvideo frame: calculating a comparison measure betweencolor-characteristics of a pixel in the raw video frame andcolor-characteristics of a corresponding pixel in a background image,wherein the comparison measure is defined in relation to a range ofpossible values of at least one color-characteristic of the pixel; upondetermining the comparison measure of the pixel is within a specifiedthreshold, classifying the pixel of the raw frame as depicting a portionof a background of the scene; and upon determining the comparisonmeasure is outside the specified threshold, classifying the pixel of theraw frame as depicting a portion of a foreground of the scene.
 2. Themethod of claim 1, wherein the specified threshold is selected based ona set of predefined brightness ranges for the at least onecolor-characteristic of the pixel, wherein the predefined brightnessranges include a range defining a dark pixel, a range defining a mediumpixel, and a range defining a bright pixel.
 3. The method of claim 1,wherein the calculating the comparison measure of the pixel in the rawvideo frame comprises: for each of a plurality of color-characteristicsof the pixel: calculating a measure of closeness between a value of acolor-characteristic of the pixel and a value of thecolor-characteristic of the corresponding pixel in the background imageusing a range of possible values of the color-characteristic.
 4. Themethod of claim 3, wherein color-characteristics of the pixel and of thecorresponding pixel in the background image are selected from a groupconsisting of a Red (R), Green (G), and Blue (B) color-channels.
 5. Themethod of claim 3, wherein calculating the measure of closeness betweencolor-characteristics of the pixel in the raw video frame andcolor-characteristics of the corresponding pixel in the backgroundimage, further comprises: determining a first value based on a range ofpossible values of a first color-channel and a difference between valuesof the first color-channel of the pixel and of the corresponding pixelin the background image; and determining a second value based on a rangeof possible values of a second color-channel and a difference betweenvalues of the second color-channel of the pixel and of the correspondingpixel in the background image, wherein the first color-channel and thesecond color-channel are different color-channels.
 6. The method ofclaim 5, wherein the calculating a measure of closeness betweencolor-characteristics of the pixel in the raw video frame andcolor-characteristics of the corresponding pixel in the backgroundimage, further comprises: calculating an average of the first value andthe second value.
 7. The method of claim 5, wherein the calculating ameasure of closeness between color-characteristics of the pixel in theraw video frame and color-characteristics of the corresponding pixel inthe background image, further comprises: determining a third value basedon a range of possible values of a third color-channel and a differencebetween values of the third color-channel of the corresponding pixel inthe background image, wherein the third color-channel is a differentcolor-channel than the first color-channel and the second color-channel;comparing the first value, the second value, and the third value todetermine a nosiest color-channel of the pixel; and calculating thecomparison measure by calculating an average of two values selected froma group consisting of the first value, the second value, and the thirdvalue, wherein a value corresponding to the noisiest color-channel isdisregarded, wherein the noisiest color-channel is a color-channelhaving a smallest value of the first value, second value, and the thirdvalue.
 8. The method of claim 1, wherein the background image includes ashort-term background image and a long-term background image.
 9. Themethod of claim 1, further comprising, training the background imageusing a set of consecutive video frames of the sequence of video frames.10. The method of claim 1, further comprising: updating the backgroundimage using color-characteristic values of each pixel of the raw videoframe classified as depicting a portion of the background of the scene.11. The method of claim 1, further comprising: for at least a firstpixel classified as depicting a portion of a foreground of the scene,determining that the first pixel was classified as depicting a portionof the foreground of the scene within a specified marginal magnitude ofthe threshold; calculating an average set of pixel color-characteristicvalues from a set of pixels neighboring the first pixel; calculating acomparison measure for the average set of pixel color-characteristicvalues and color-characteristics of a corresponding pixel in abackground image; and upon determining the comparison measure for theaverage set of pixel color-characteristic values is within the specifiedthreshold for classing a pixel as depicting scene background,re-classifying the first pixel of the raw frame as depicting a portionof a background of the scene.
 12. A computer-readable storage mediumcontaining a program which, when executed by a processor, performs anoperation for analyzing a scene depicted in a sequence of video frames,the operation comprising: receiving a raw video frame of the sequence ofvideo frames; and for each of a plurality of pixels of the raw videoframe: calculating a comparison measure between color-characteristics ofa pixel in the raw video frame and color-characteristics of acorresponding pixel in a background image, wherein the comparisonmeasure is defined in relation to a range of possible values of at leastone color-characteristic of the pixel, upon determining the comparisonmeasure of the pixel is within a specified threshold, classifying thepixel of the raw frame as depicting a portion of a background of thescene, and upon determining the comparison measure is outside thespecified threshold, classifying the pixel of the raw frame as depictinga portion of a foreground of the scene.
 13. The computer-readablestorage medium of claim 12, wherein the specified threshold is selectedbased on a set of predefined brightness-ranges for the at least onecolor-characteristic of the pixel, wherein the predefined brightnessranges include a range defining a dark pixel, a range defining a mediumpixel, and a range defining a bright pixel.
 14. The computer-readablestorage medium of claim 12, wherein the calculating the comparisonmeasure of the pixel in the raw video frame comprises: for each of aplurality of color-characteristics of the pixel: calculating a measureof closeness between a value of a color-characteristic of the pixel anda value of the color-characteristic of the corresponding pixel in thebackground image using a range of possible values of thecolor-characteristic.
 15. The computer-readable storage medium of claim14, wherein calculating the measure of closeness betweencolor-characteristics of the pixel in the raw video frame andcolor-characteristics of the corresponding pixel in the backgroundimage, further comprises: determining a first value based on a range ofpossible values of a first color-channel and a difference between valuesof the first color-channel of the pixel and of the corresponding pixelin the background image; and determining a second value based on a rangeof possible values of a second color-channel and a difference betweenvalues of the second color-channel of the pixel and of the correspondingpixel in the background image, wherein the first color-channel and thesecond color-channel are different color-channels.
 16. Thecomputer-readable storage medium of claim 15, wherein the calculating ameasure of closeness between color-characteristics of the pixel in theraw video frame and color-characteristics of the corresponding pixel inthe background image, further comprises: calculating an average of thefirst value and the second value.
 17. The computer-readable storagemedium of claim 15, wherein the calculating a measure of closenessbetween color-characteristics of the pixel in the raw video frame andcolor-characteristics of the corresponding pixel in the backgroundimage, further comprises: determining a third value based on a range ofpossible values of a third color-channel and a difference between valuesof the third color-channel of the corresponding pixel in the backgroundimage, wherein the third color-channel is a different color-channel thanthe first color-channel and the second color-channel; comparing thefirst value, the second value, and the third value to determine anosiest color-channel of the pixel; and calculating the comparisonmeasure by calculating an average of two values selected from a groupconsisting of the first value, the second value, and the third value,wherein a value corresponding to the noisiest color-channel isdisregarded.
 18. The computer-readable storage medium of claim 12,wherein the operation further comprises, training the background imageusing a set of consecutive video frames of the sequence of video frames.19. The computer-readable storage medium of claim 12, wherein theoperation further comprises updating the background image usingcolor-characteristic values of each pixel of the raw video frameclassified as depicting a portion of the background of the scene. 20.The computer-readable storage medium of claim 12, wherein the operationfurther comprises: for at least a first pixel classified as depicting aportion of a foreground of the scene, determining that the first pixelwas classified as depicting a portion of the foreground of the scenewithin a specified marginal magnitude of the threshold; calculating anaverage set of pixel color-characteristic values from a set of pixelsneighboring the first pixel; calculating a comparison measure for theaverage set of pixel color-characteristic values andcolor-characteristics of a corresponding pixel in a background image;and upon determining the comparison measure for the average set of pixelcolor-characteristic values is within the specified threshold forclassing a pixel as depicting scene background, re-classifying the firstpixel of the raw frame as depicting a portion of a background of thescene.
 21. A system comprising, a video input source configured toprovide a sequence of video frames, each depicting a scene; a processor;and a memory containing a video analysis application, which whenexecuted by the processor is configured to perform an operation foranalyzing the scene depicted in the sequence of video frames, theoperation comprising: receiving a raw video frame from the video inputsource, and for each of a plurality of pixels of the raw video frame:calculating a comparison measure between color-characteristics of apixel in the raw video frame and color-characteristics of acorresponding pixel in a background image, wherein the comparisonmeasure is defined in relation to a range of possible values of at leastone color-characteristic of the pixel, upon determining the comparisonmeasure of the pixel is within a specified threshold, classifying thepixel of the raw frame as depicting a portion of a background of thescene, and upon determining the comparison measure is outside thespecified threshold, classifying the pixel of the raw frame as depictinga portion of a foreground of the scene.
 22. The system of claim 21,wherein the specified threshold is selected based on a set of predefinedbrightness ranges for the at least one color-characteristic of thepixel, wherein the predefined brightness ranges include a range defininga dark pixel, a range defining a medium pixel, and a range defining abright pixel.
 23. The system of claim 21, wherein the operation furthercomprises, training the background image using a set of consecutivevideo frames of the sequence of video frames.
 24. The system of claim21, wherein the operation further comprises: updating the backgroundimage using color-characteristic values of each pixel of the raw videoframe classified as depicting a portion of the background of the scene.25. The system of claim 21, wherein the operation further comprises: forat least a first pixel classified as depicting a portion of a foregroundof the scene, determining that the first pixel was classified asdepicting a portion of the foreground of the scene within a specifiedmarginal magnitude of the threshold; calculating an average set of pixelcolor-characteristic values from a set of pixels neighboring the firstpixel; calculating a comparison measure for the average set of pixelcolor-characteristic values and color-characteristics of a correspondingpixel in a background image; and upon determining the comparison measurefor the average set of pixel color-characteristic values is within thespecified threshold for classing a pixel as depicting scene background,re-classifying the first pixel of the raw frame as depicting a portionof a background of the scene.